Apparatus and method for authenticating user in augmented reality

ABSTRACT

An electronic device is provided. The electronic device obtains current spatial information on a physical space where a user is positioned using a sensor, generates a greater number of virtual objects than the number of unavailable authentication objects based on at least one authentication object among a plurality of authentication objects being unavailable in the current spatial information, outputs the generated virtual objects to a plurality of positions including a physical position of an unavailable authentication object, and determines that authentication is successful based on a user input that selects a generated virtual object and an available authentication object as a registered sequence.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application, claiming priority under§ 365(c), of an International application No. PCT/KR2023/007773, filedon Jun. 7, 2023, which is based on and claims the benefit of a Koreanpatent application number 10-2022-0096834, filed on Aug. 3, 2022, in theKorean Intellectual Property Office, and of a Korean patent applicationnumber 10-2022-0109857, filed on Aug. 31, 2022, in the KoreanIntellectual Property Office, the disclosure of each of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an apparatus and method for authenticating auser in augmented reality.

2. Description of Related Art

With the development of information communication technology andsemiconductor technology, various electronic devices are developing intomultimedia devices that provide various multimedia services. Amultimedia service may include any one or any combination of a voicecall service, a message service, a broadcast service, a wirelessInternet service, a camera service, an electronic payment service, and amusic playback service.

An electronic device may provide experiences to users in various forms.For example, an electronic device may provide users with various typesof experiences based on an augmented reality (AR) service in whichvirtual information (e.g., virtual objects) is added to a real space. Anelectronic device may perform user authentication in AR.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to provideto an apparatus and method for authenticating a user in augmentedreality.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic device isprovided. The electronic device includes a display that providesaugmented reality, a sensor that collects information for providingaugmented reality, a memory that stores computer-executableinstructions, and a processor that executes the instructions byaccessing the memory. The instructions may include obtaining currentspatial information on a physical space where a user is positioned usinga sensor. The instructions may include generating a greater number ofvirtual objects than the number of unavailable authentication objectsbased on at least one authentication object among a plurality ofauthentication objects being unavailable in the current spatialinformation. The instructions may include outputting the generatedvirtual objects to a plurality of positions including a physicalposition of an unavailable authentication object. The instructions mayinclude determining that authentication is successful based on a userinput that selects the generated virtual object and an availableauthentication object as a registered sequence.

In accordance with another aspect of the disclosure, a methodimplemented by a processor is provided. The method includes obtainingcurrent spatial information on a physical space where a user ispositioned using a sensor that collects information for providingaugmented reality. The method may include generating a greater number ofvirtual objects than the number of unavailable authentication objectsbased on at least one authentication object among a plurality ofauthentication objects being unavailable in the current spatialinformation. The method may include outputting the generated virtualobjects to a plurality of positions including a physical position of anunavailable authentication object using a display that providesaugmented reality. The method may include determining thatauthentication is successful based on a user input that selects thegenerated virtual object and an available authentication object as aregistered sequence.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram illustrating a configuration of an electronicdevice in a network environment, according to an embodiment of thedisclosure;

FIG. 2 is a diagram illustrating a structure of an electronic deviceaccording to an embodiment of the disclosure;

FIG. 3 is a diagram illustrating an operation of an eye tracking cameraincluded in an electronic device according to an embodiment of thedisclosure;

FIGS. 4 and 5 are flowcharts illustrating a user authentication methodaccording to various embodiments of the disclosure;

FIG. 6 is a diagram illustrating an example of obtaining current spatialinformation according to an embodiment of the disclosure;

FIG. 7 is a diagram illustrating an operation of selecting anauthentication object according to an embodiment of the disclosure;

FIG. 8 is a diagram illustrating types of authentication objectsaccording to an embodiment of the disclosure;

FIG. 9 is a diagram illustrating an operation of selecting anauthentication object according to an authentication sequence accordingto an embodiment of the disclosure;

FIG. 10 is a diagram illustrating the generation of a virtual objectwhen some authentication objects are unavailable according to anembodiment of the disclosure; and

FIG. 11 is a flowchart illustrating an operation of registering a newauthentication sequence according to an embodiment of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 1 is a block diagram illustrating an electronic device in a networkenvironment according to an embodiment of the disclosure.

Referring to FIG. 1 , an electronic device 101 in a network environment100 may communicate with an electronic device 102 via a first network198 (e.g., a short-range wireless communication network), or communicatewith at least one of an electronic device 104 or a server 108 via asecond network 199 (e.g., a long-range wireless communication network).According to an embodiment, the electronic device 101 may communicatewith the electronic device 104 via the server 108. According to anembodiment, the electronic device 101 may include a processor 120, amemory 130, an input module 150, a sound output module 155, a displaymodule 160, an audio module 170, and a sensor module 176, an interface177, a connecting terminal 178, a haptic module 179, a camera module180, a power management module 188, a battery 189, a communicationmodule 190, a subscriber identification module (SIM) 196, or an antennamodule 197. In some embodiments, at least one (e.g., the connectingterminal 178) of the components may be omitted from the electronicdevice 101, or one or more other components may be added in theelectronic device 101. In some embodiments, some (e.g., the sensormodule 176, the camera module 180, or the antenna module 197) of thecomponents may be integrated as a single component (e.g., the displaymodule 160).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 connected to theprocessor 120, and may perform various data processing or computation.According to an embodiment, as at least a part of data processing orcomputation, the processor 120 may store a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in a volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data in anon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), or an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), a neural processing unit (NPU), animage signal processor (ISP), a sensor hub processor, or a communicationprocessor (CP)) that is operable independently from, or in conjunctionwith the main processor 121. For example, when the electronic device 101includes the main processor 121 and the auxiliary processor 123, theauxiliary processor 123 may be adapted to consume less power than themain processor 121 or to be specific to a specified function. Theauxiliary processor 123 may be implemented separately from the mainprocessor 121 or as a part of the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one of the components (e.g., the displaymodule 160, the sensor module 176, or the communication module 190) ofthe electronic device 101, instead of the main processor 121 while themain processor 121 is in an inactive (e.g., sleep) state or along withthe main processor 121 while the main processor 121 is an active state(e.g., executing an application). According to an embodiment, theauxiliary processor 123 (e.g., an ISP or a CP) may be implemented as aportion of another component (e.g., the camera module 180 or thecommunication module 190) that is functionally related to the auxiliaryprocessor 123. According to an embodiment, the auxiliary processor 123(e.g., an NPU) may include a hardware structure specified for artificialintelligence (AI) model processing. An AI model may be generated throughmachine learning. Such learning may be performed by, for example, theelectronic device 101, in which artificial intelligence is performed, orperformed via a separate server (e.g., the server 108). Learningalgorithms may include, but are not limited to, for example, supervisedlearning, unsupervised learning, semi-supervised learning, orreinforcement learning algorithms. The AI model may include a pluralityof artificial neural network layers. An artificial neural network mayinclude, for example, a deep neural network (DNN), a convolutionalneural network (CNN), a recurrent neural network (RNN), a restrictedBoltzmann machine (RBM), a deep belief network (DBN), a bidirectionalrecurrent deep neural network (BRDNN), a deep Q-network, or acombination of two or more thereof, but is not limited thereto. The AImodel may additionally or alternatively include a software structureother than the hardware structure.

The memory 130 may store various pieces of data used by at least onecomponent (e.g., the processor 120 or the sensor module 176) of theelectronic device 101. The data may include, for example, software(e.g., the program 140) and input data or output data for a commandrelated thereto. The memory 130 may include the volatile memory 132 orthe non-volatile memory 134.

The program 140 may be stored as software in the memory 130, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input module 150 may receive a command or data to be used by anothercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputmodule 150 may include, for example, a microphone, a mouse, a keyboard,a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 155 may output a sound signal to the outside ofthe electronic device 101. The sound output module 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing a recording. Thereceiver may be used to receive an incoming call. According to anembodiment, the receiver may be implemented separately from the speakeror as a part of the speaker.

The display module 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display module 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display, thehologram device, and the projector. According to an embodiment, thedisplay module 160 may include a touch sensor adapted to sense a touch,or a pressure sensor adapted to measure an intensity of a force incurredby the touch.

The audio module 170 may convert a sound into an electric signal or viceversa. According to an embodiment, the audio module 170 may acquire thesound via the input module 150 or output the sound via the sound outputmodule 155 or an external electronic device (e.g., the electronic device102 such as a speaker or headphones) directly or wirelessly connected tothe electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andgenerate an electric signal or data value corresponding to the detectedstate. According to an embodiment, the sensor module 176 may include,for example, a gesture sensor, a gyro sensor, an atmospheric pressuresensor, a magnetic sensor, an acceleration sensor, a grip sensor, aproximity sensor, a color sensor, an infrared (IR) sensor, a biometricsensor, a temperature sensor, a humidity sensor, or an illuminancesensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., by wire) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high-definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

The connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected to an externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, anHDMI connector, a USB connector, an SD card connector, or an audioconnector (e.g., a headphones connector).

The haptic module 179 may convert an electric signal into a mechanicalstimulus (e.g., a vibration or a movement) or an electrical stimuluswhich may be recognized by a user via their tactile sensation orkinesthetic sensation. According to an embodiment, the haptic module 179may include, for example, a motor, a piezoelectric element, or anelectric stimulator.

The camera module 180 may capture a still image and moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, ISPs, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to an embodiment, the power managementmodule 188 may be implemented as, for example, at least a part of apower management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more CPs that are operableindependently from the processor 120 (e.g., an AP) and that supportdirect (e.g., wired) communication or wireless communication. Accordingto an embodiment, the communication module 190 may include a wirelesscommunication module 192 (e.g., a cellular communication module, ashort-range wireless communication module, or a global navigationsatellite system (GNSS) communication module) or a wired communicationmodule 194 (e.g., a local area network (LAN) communication module, or apower line communication (PLC) module). A corresponding one of thesecommunication modules may communicate with the external electronicdevice 104 via the first network 198 (e.g., a short-range communicationnetwork, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, orinfrared data association (IrDA)) or the second network 199 (e.g., along-range communication network, such as a legacy cellular network, afifth generation (5G) network, a next-generation communication network,the Internet, or a computer network (e.g., a LAN or a wide area network(WAN))). These various types of communication modules may be implementedas a single component (e.g., a single chip), or may be implemented asmultiple components (e.g., multiple chips) separate from each other. Thewireless communication module 192 may identify and authenticate theelectronic device 101 in a communication network, such as the firstnetwork 198 or the second network 199, using subscriber information(e.g., international mobile subscriber identity (IMSI)) stored in theSIM 196.

The wireless communication module 192 may support a 5G network after a4G network, and next-generation communication technology, for example,new radio (NR) access technology. The NR access technology may supportenhanced mobile broadband (eMBB), massive machine type communications(mMTC), or ultra-reliable and low-latency communications (URLLC). Thewireless communication module 192 may support a high-frequency band(e.g., a millimeter-wave (mmWave) band) to achieve, for example, a highdata transmission rate. The wireless communication module 192 maysupport various technologies for securing performance on ahigh-frequency band, such as, e.g., beamforming, massive multiple-inputand multiple-output (MIMO), full dimensional MIMO (FD-MIMO), an arrayantenna, analog beam-forming, or a large-scale antenna. The wirelesscommunication module 192 may support various requirements specified inthe electronic device 101, an external electronic device (e.g., theelectronic device 104), or a network system (e.g., the second network199). According to an embodiment, the wireless communication module 192may support a peak data rate (e.g., 20 gigabits per second (Gbps) ormore) for implementing eMBB, loss coverage (e.g., 164 decibel (dB) orless) for implementing mMTC, or U-plane latency (e.g., 0.5 millisecond(ms) or less for each of downlink (DL) and uplink (UL), or a round tripof 1 ms or less) for implementing URLLC.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., an external electronic device) of the electronicdevice 101. According to an embodiment, the antenna module 197 mayinclude an antenna including a radiating element including a conductivematerial or a conductive pattern formed in or on a substrate (e.g., aprinted circuit board (PCB)). According to an embodiment, the antennamodule 197 may include a plurality of antennas (e.g., an antenna array).In such a case, at least one antenna appropriate for a communicationscheme used in a communication network, such as the first network 198 orthe second network 199, may be selected by, for example, thecommunication module 190 from the plurality of antennas. The signal orpower may be transmitted or received between the communication module190 and the external electronic device via the at least one selectedantenna. According to an embodiment, another component (e.g., a radiofrequency integrated circuit (RFIC)) other than the radiating elementmay be additionally formed as a part of the antenna module 197.

According to an embodiment, the antenna module 197 may form a mmWaveantenna module. According to an embodiment, the mmWave antenna modulemay include a printed circuit board, an RFIC disposed on a first surface(e.g., the bottom surface) of the printed circuit board, or adjacent tothe first surface and capable of supporting a designated high-frequencyband (e.g., the mmWave band), and a plurality of antennas (e.g., arrayantennas) disposed on a second surface (e.g., the top or a side surface)of the printed circuit board, or adjacent to the second surface andcapable of transmitting or receiving signals of the designatedhigh-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the external electronic devices 102 or 104 may be a device of thesame type as or a different type from the electronic device 101.According to an embodiment, all or some of operations to be executed bythe electronic device 101 may be executed at one or more externalelectronic devices (e.g., the external electronic devices 102 and 104,and the server 108). For example, if the electronic device 101 needs toperform a function or a service automatically, or in response to arequest from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request one or more external electronic devices to perform at leastpart of the function or the service. The one or more external electronicdevices receiving the request may perform the at least part of thefunction or the service requested, or an additional function or anadditional service related to the request, and may transfer a result ofthe performing to the electronic device 101. The electronic device 101may provide the result, with or without further processing of theresult, as at least part of a response to the request.

For example, the external electronic device 102 may render content dataexecuted by an application and then transmit the data to the electronicdevice 101, and the electronic device 101 receiving the data may outputthe content data to the display module 160. If the electronic device 101detects a motion of a user through an inertial measurement unit (IMU)sensor, and the like, the processor 120 of the electronic device 101 maycorrect the rendered data received from the external electronic device102 based on information on the motion and output the corrected data tothe display module 160. Alternatively, the processor may transmit theinformation on the motion to the external electronic device 102 and senda rendering request such that screen data is updated accordingly.According to an embodiment, the external electronic device 102 may be adevice in various forms, such as a smartphone or a case device forstoring and charging the electronic device 101.

To that end, cloud computing, distributed computing, mobile edgecomputing (MEC), or client-server computing technology may be used, forexample. The electronic device 101 may provide ultra low-latencyservices using, for example, distributed computing or mobile edgecomputing. In an embodiment, the external electronic device 104 mayinclude an Internet-of-things (IoT) device. The server 108 may be anintelligent server using machine learning and/or a neural network.According to an embodiment, the external electronic device 104 or theserver 108 may be included in the second network 199. The electronicdevice 101 may be applied to intelligent services (e.g., a smart home, asmart city, a smart car, or healthcare) based on 5G communicationtechnology or IoT-related technology.

FIG. 2 is a diagram illustrating a structure of an electronic deviceaccording to an embodiment of the disclosure.

Referring to FIG. 2 , a wearable electronic device 200 (e.g., theelectronic device 101 or 102 of FIG. 1 ) may be worn on a face of a userto provide the user with an image associated with an augmented reality(AR) service and/or a virtual reality (VR) service.

In an embodiment, the wearable electronic device 200 may include a firstdisplay 205, a second display 210, screen display portions 215 a and 215b, input optical members 220 a and 220 b, a first transparent member 225a, a second transparent member 225 b, lighting units 230 a and 230 b, afirst PCB 235 a, a second PCB 235 b, a first hinge 240 a, a second hinge240 b, an imaging camera 245, a plurality of microphones (e.g., a firstmicrophone 250 a, a second microphone 250 b, and a third microphone 250c), a plurality of speakers (e.g., a first speaker 255 a and a secondspeaker 255 b), a battery 260, a first recognition camera 265 a, asecond recognition camera 265 b, a first eye tracking camera 270 a, anda second eye tracking camera 270 b.

In an embodiment, a display (e.g., the first display 205 and the seconddisplay 210) may include, for example, a liquid crystal display (LCD), adigital mirror device (DMD), a liquid crystal on silicon (LCoS), anorganic light-emitting diode (OLED), or a micro light-emitting diode(micro LED). Although not shown in the drawings, when the first display205 and/or the second display 210 is one of an LCD, a DMD, or a LCoS,the wearable electronic device 200 may include a light source whichemits light to a screen output area of the first display 205 and/or thesecond display 210. In an embodiment, when the first display 205 and/orthe second display 210 is capable of generating light by itself (whenthe first display 205 and/or the second display 210 is either an OLED ora micro-LED, for example), the wearable electronic device 200 mayprovide a virtual image with a relatively high quality to the user eventhough a separate light source is not included. For example, when thefirst display 205 and/or the second display 210 is implemented as anOLED or a micro-LED, a light source may be unnecessary, and accordinglythe wearable electronic device 200 may be lightened. Hereinafter, thefirst display 205 and/or the second display 210 capable of generatinglight by itself may be referred to as a “self-luminous display,” anddescription will be made on the assumption of the self-luminous display.

The first display 205 and/or the second display 210 according to anembodiment may include at least one micro-LED. For example, themicro-LED may express red (R), green (G), and blue (B) by emitting lightby itself, and a single chip may implement a single pixel (e.g., one ofR, G, and B pixels) because the micro-LED is relatively small in size(e.g., 100 μm or less). Accordingly, it may be possible to provide ahigh resolution without a backlight unit (BLU), when the first display205 and/or the second display 210 is implemented as a micro-LED.However, the embodiments are not limited thereto, and a single chip maybe implemented by a plurality of pixels including R, G, and B pixels.The first display 205 and/or the second display 210 may also be referredto as a “light source.”

In an embodiment, the first display 205 and/or the second display 210may include pixels for displaying a virtual image. The first display 205and/or the second display 210 may further include infrared pixels thatemit infrared light.

In an embodiment, the first display 205 and/or the second display 210may further include light-receiving pixels (e.g., photo sensor pixels)that are disposed between pixels, receive light reflected from eyes of auser, convert the received light to electrical energy, and output theelectrical energy. A light-receiving pixel(s) may be referred to as an“eye tracking sensor.” The eye tracking sensor (e.g., an eye trackingsensor 315 of FIG. 3 ) may sense infrared light generated by reflectinglight emitted by an infrared pixel included in the first display 205and/or the second display 210 by eyes of a user.

The wearable electronic device 200 may detect a gaze direction (e.g., amovement of a pupil) of the user, using light-receiving pixels (e.g.,eye tracking sensor 315). For example, the wearable electronic device200 may detect and track a gaze direction of each of a right eye and aleft eye of the user through one or more light-receiving pixels (e.g.,eye tracking sensor 315) of the first display 205 and one or morelight-receiving pixels (e.g., eye tracking sensor 315) of the seconddisplay 210. The wearable electronic device 200 may also determine acentral position of a virtual image according to the gaze directions ofthe right eye and the left eye of the user (e.g., directions in whichpupils of the right eye and the left eye of the user gaze) detectedthrough the one or more light-receiving pixels (e.g., eye trackingsensor 315).

The wearable electronic device 200 may include the first display 205and/or the second display 210, the first transparent member 225 a and/orthe second transparent member 225 b. A user may use the wearableelectronic device 200 while wearing the wearable electronic device 200on a face of the user. In an embodiment, the first transparent member225 a may face the right eye of the user, and the second transparentmember 225 b may face the left eye of the user. In an embodiment, whenthe first display 205 and/or the second display 210 are transparent, thefirst display 205 and/or the second display 210 may face the eyes of theuser to configure the screen display portions 215 a and 215 b.

The first display 205 and the second display 210 may each include afirst control circuit (not shown). The first control circuit may controlthe first display 205 and the second display 210. The first controlcircuit may control an operation of a liquid crystal element of atransparent cover (not shown) included in each of the first display 205and the second display 210. In an embodiment, light emitted from thefirst display 205 and/or the second display 210 may reach the screendisplay portion 215 a formed on the first transparent member 225 a thatfaces the right eye of the user, and the screen display portion 215 bformed on the second transparent member 225 b that faces the left eye ofthe user, by passing through a lens (not shown) and a waveguide (e.g., adisplay waveguide 350 and an eye tracking waveguide 360 of FIG. 3 ).

The lens (not shown) may be disposed in front of the first display 205and/or the second display 210. The lens (not shown) may include aconcave lens and/or a convex lens. For example, the lens (not shown) mayinclude a projection lens (e.g., a projection lens 325 of FIG. 3 ), or acollimation lens (not shown).

In an embodiment, the light emitted from the first display 205 and/orthe second display 210 may be guided by the display waveguide 350 and/orthe eye tracking waveguide 360 through the input optical members 220 aand 220 b. Light moving into the display waveguide 350 and/or the eyetracking waveguide 360 may be guided toward eyes of a user through anoutput optical member (e.g., an output optical member 340 of FIG. 3 ).The screen display portions 215 a and 215 b may be determined based onlight emitted toward an eye of a user (e.g., an eye 301 of the user ofFIG. 3 ).

For example, the light emitted from the first display 205 and/or thesecond display 210 may be reflected from a grating area of the displaywaveguide 350 and/or the eye tracking waveguide 360 formed in the inputoptical member 220 a, 220 b and the screen display portion 215 a, 215 b,and may be transmitted to the eye 301 of the user.

In an embodiment, the screen display portions 215 a and 215 b or atransparent member (e.g., the first transparent member 225 a and thesecond transparent member 225 b) may include a reflective lens, and alens including the display waveguide 350 and/or the eye trackingwaveguide 360. The display waveguide 350 and the eye tracking waveguide360 may function to transmit a light source generated by the firstdisplay 205 and/or the second display 210 to eyes of the user, and maybe referred to as an “optical waveguide.” Hereinafter, an “opticalwaveguide” or “wave guide” may correspond to the screen display portions215 a and 215 b.

The screen display portions 215 a and 215 b may be a path through whichexternal light is incident, totally reflected, and emitted, and may bedistinguished from the first transparent member 225 a and the secondtransparent member 225 b through which external light is simplyreflected or transmitted.

In an embodiment, the screen display portions 215 a and 215 b mayinclude glass, plastic, or a polymer, and may have a nanopattern formedon one surface of the inside or outside, that is, a grating structure ofa polygonal or curved shape. In an embodiment, light incident to one endof the screen display portions 215 a and 215 b through the input opticalmembers 220 a and 220 b may be propagated inside the display waveguide350 by the nanopattern to be provided to the user. For example, thescreen display portions 215 a and 215 b including a freeform prism mayprovide incident light to a user through a reflection mirror.

The screen display portions 215 a and 215 b may include at least one ofa reflective element (e.g., a reflection mirror) and at least onediffractive element (e.g., a diffractive optical element (DOE) or aholographic optical element (HOE)). The screen display portions 215 aand 215 b may guide light emitted from a display (e.g., the firstdisplay 205 and the second display 210) to the eyes of the user, usingthe at least one diffractive element or the reflective element includedin the screen display portions 215 a and 215 b.

In an embodiment, the diffractive element may include the input opticalmembers 220 a and 220 b and/or an output optical member (e.g., theoutput optical member 340 of FIG. 3 ). For example, the input opticalmembers 220 a and 220 b may refer to an input grating area, and theoutput optical member 340 may refer to an output grating area. The inputgrating area may function as an input terminal to diffract (or reflect)light output from the first display 205 and/or the second display 210(e.g., a micro LED) to transmit the light to the screen display portions215 a and 215 b. The output grating area may function as an exit todiffract (or reflect) light transmitted to the display waveguide 350and/or the eye tracking waveguide 360 to the eye 301 of the user.

In an embodiment, the reflective element may include a total internalreflection optical element or a total internal reflection waveguide fortotal internal reflection (TIR). For example, total reflection, which isone of schemes of inducing light, may define an angle of incidence suchthat light (e.g., a virtual image) entering through the input gratingarea is completely or almost completely reflected from a portion (e.g.,a predetermined surface) of the screen display portions 215 a and 215 b,to completely or almost completely transmit the light to the outputgrating area.

The first transparent member 225 a and/or the second transparent member225 b may be formed as a glass plate, a plastic plate, or a polymer, andmay be transparently or translucently formed, for example. In anembodiment, the first transparent member 225 a may face the right eye ofthe user, and the second transparent member 225 b may face the left eyeof the user.

The lighting units 230 a and 230 b may be used differently according topositions in which the lighting units 230 a and 230 b are attached. Forexample, the lighting units 230 a and 230 b may be attached around aframe of the wearable electronic device 200. The lighting units 230 aand 230 b may be used as auxiliary devices for facilitating eye-gazedetection when pupils are captured using the first eye tracking camera270 a and the second eye tracking camera 270 b. The lighting units 230 aand 230 b may use an IR LED with a visible light wavelength or aninfrared light wavelength.

Alternatively, the lighting units 230 a and 230 b may be attached arounda hinge (e.g., the first hinge 240 a and the second hinge 240 b) thatconnects a frame (e.g., a rim) and a temple corresponding to a legportion of glasses of the wearable electronic device 200 or around acamera (e.g., the first recognition camera 265 a and the secondrecognition camera 265 b) mounted adjacent to a bridge that connects theframe. Here, the first recognition camera 265 a and the secondrecognition camera 265 b may be, for example, global shutter (GS)cameras, but are not limited thereto.

When capturing is performed using a GS camera, the lighting units 230 aand 230 b may be used to supplement a surrounding brightness. Forexample, the lighting units 230 a and 230 b may be used in a darkenvironment or when it is not easy to detect a subject to be captureddue to reflected light and mixing of various light sources.

In an embodiment, the lighting units 230 a and 230 b may be omitted. Thelighting units 230 a and 230 b may be replaced by infrared pixelsincluded in the first display 205 and the second display 210. In anembodiment, the lighting units 230 a and 230 b may be included in thewearable electronic device 200 to assist infrared pixels included in thefirst display 205 and the second display 210.

A PCB (e.g., the first PCB 235 a and the second PCB 235 b) may bedisposed in the temple portion of the wearable electronic device 200 andmay transmit an electrical signal to each module (e.g., a camera, adisplay, an audio, or a sensor) and another PCB through a flexibleprinted circuit board (FPCB). In an embodiment, at least one PCB mayinclude the first PCB 235 a, the second PCB 235 b, and an interposer(not shown) disposed between the first PCB 235 a and the second PCB 235b.

In an embodiment, a control circuit (not shown) for controllingcomponents of the wearable electronic device 200 other than the firstdisplay 205 and the second display 210 may be disposed on a PCB (e.g.,the first PCB 235 a and the second PCB 235 b). The control circuit maycontrol components other than the first display 205 and the seconddisplay 210 and perform an operation such as depth value estimation. Thecontrol circuit may include a communication circuit (e.g., thecommunication module 190 of FIG. 1 ) or a memory (e.g., the memory 130of FIG. 1 ). The control circuit may control the first display 205, thesecond display 210, and/or the other components.

The first hinge 240 a and/or the second hinge 240 b may correspond to aportion where the temple and the frame (e.g., the rim) of the wearableelectronic device 200 are coupled.

In an embodiment, the imaging camera 245 may be referred to as a “highresolution (HR)” or a “photo video (PV),” and may include ahigh-resolution camera. The imaging camera 245 may include a colorcamera having functions for obtaining a high-quality image, such as anautomatic focus (AF) function and an optical image stabilizer (OIS). Theembodiments are not limited thereto, and the imaging camera 245 mayinclude a GS camera or a rolling shutter (RS) camera.

In an embodiment, a plurality of microphones (e.g., the first microphone250 a, the second microphone 250 b, and the third microphone 250 c) mayconvert an external acoustic signal into electrical audio data. Theelectrical audio data may be variously utilized according to a function(or an application being executed) being performed by the wearableelectronic device 200.

In an embodiment, a plurality of speakers (e.g., the first speaker 255 aand the second speaker 255 b) may output audio data that is receivedfrom a communication circuit (e.g., the communication module 190 of FIG.1 ) or stored in a memory (e.g., the memory 130 of FIG. 1 ).

In an embodiment, one or more batteries 260 may be included, and maysupply power to components constituting the wearable electronic device200.

In an embodiment, the first recognition camera 265 a and the secondrecognition camera 265 b may include cameras used for three degrees offreedom (3DoF) and six degrees of freedom (6DoF) head tracking, handdetection and tracking, and gesture and/or space recognition. Forexample, the first recognition camera 265 a and the second recognitioncamera 265 b may each include a GS camera to detect a movement of a heador a hand and track the movement. For example, a stereo camera may beused for head tracking and space recognition, and accordingly two GScameras with the same standard and performance may be used. An RS cameramay be used to detect a quick hand movement and a minute movement of afinger and track a movement. In an embodiment, a GS camera havingsuperior performance (e.g., image drag) in comparison to a camera may bemainly used, however, the embodiments are not limited thereto. In anembodiment, an RS camera may also be used. The first recognition camera265 a and the second recognition camera 265 b may perform a simultaneouslocalization and mapping (SLAM) function through depth capturing andspatial recognition for 6DoF. In addition, the first recognition camera265 a and the second recognition camera 265 b may perform a user gesturerecognition function.

In an embodiment, at least one sensor (not shown, e.g., a gyro sensor,an acceleration sensor, a geomagnetic sensor, and/or a gesture sensor),the first recognition camera 265 a, and the second recognition camera265 b may perform at least one of head tracking for 6DoF, poseestimation and prediction, gesture and/or space recognition, and/or afunction of a SLAM through depth imaging.

In an embodiment, the first recognition camera 265 a and the secondrecognition camera 265 b may be classified and used as a camera for headtracking and a camera for hand tracking.

In an embodiment, the first eye tracking camera 270 a and the second eyetracking camera 270 b may detect and track pupils. The first eyetracking camera 270 a and the second eye tracking camera 270 b may beused to allow a center of a virtual image projected onto the wearableelectronic device 200 to be disposed based on a direction in which apupil of a user wearing the wearable electronic device 200 gazes. Forexample, as the first eye tracking camera 270 a and the second eyetracking camera 270 b, a GS camera may be mainly used to detect a pupiland track a fast pupil movement. The first eye tracking camera 270 a maybe installed to correspond to the right eye of the user, and the secondeye tracking camera 270 b may be installed to correspond to the left eyeof the user. Here, the first eye tracking camera 270 a and the secondeye tracking camera 270 b may have the same camera performance andspecifications, however, the embodiments are not limited thereto. Anoperation of an eye tracking camera (e.g., the first eye tracking camera270 a and the second eye tracking camera 270 b) will be described inmore detail below with reference to FIG. 3 .

FIG. 3 is a diagram illustrating an operation of an eye tracking cameraincluded in an electronic device according to an embodiment of thedisclosure.

Referring to FIG. 3 , FIG. 3 illustrates a process in which an eyetracking camera 310 (e.g., the first eye tracking camera 270 a and thesecond eye tracking camera 270 b of FIG. 2 ) of a wearable electronicdevice 300 according to an embodiment tracks the eye 301 of the user,that is, a gaze of the user, using light (e.g., infrared light) outputfrom a display 320 (e.g., the first display 205 and the second display210 of FIG. 2 ).

The eye tracking camera 310 may include the eye tracking sensor 315. Theeye tracking sensor 315 may be included inside the eye tracking camera310. The eye tracking sensor 315 may detect first reflected light thatis generated when reflected infrared light 303 is reflected from the eye301 of the user. The eye tracking camera 310 may track the eye 301 ofthe user, that is, the gaze of the user, based on a detection result ofthe eye tracking sensor 315.

The display 320 may include a plurality of visible light pixels and aplurality of infrared pixels. The visible light pixels may include R, G,and B pixels. The visible light pixels may output visible lightcorresponding to a virtual object image. The infrared pixels may outputinfrared light. The display 320 may include, for example, micro LEDs, orOLEDs.

The wearable electronic device 300 may perform gaze tracking using theinfrared light output from the display 320. The projection lens 325 maybe disposed between the display 320 and an input optical member 330(e.g., the input optical members 220 a and 220 b of FIG. 2 ).

The infrared light output from the display 320 may be incident on theinput optical member 330 through the projection lens 325, and may beseparated into the reflected infrared light 303 and transmitted infraredlight 305 by a half mirror (not shown) included in the input opticalmember 330.

The half mirror may be formed in the entire area or a partial area ofthe input optical member 330. When the half mirror is formed in theentire area of the input optical member 330, the input optical member330 may also be referred to as a “half mirror.” The half mirror may bedisposed in the input optical member 330 of the display waveguide 350.The half mirror may be disposed inside or below the input optical member330. The half mirror may include a grating structure.

The half mirror may output reflected infrared light and transmittedinfrared light in response to the infrared light output from the display320. The half mirror may include a grating structure. The gratingstructure may output reflected infrared light directed toward the eye301 of the user by reflecting a portion of the output infrared light, ormay output the reflected infrared light 303 toward the eye 301 of theuser through the output optical member 340 by passing through thedisplay waveguide 350. Also, the grating structure may output thetransmitted infrared light 305 by transmitting another portion of theoutput infrared light.

The reflected infrared light 303 may be output directly toward the eye301 of the user. The reflected infrared light 303 may be output towardthe eye 301 of the user through the output optical member 340 by passingthrough the display waveguide 350. The transmitted infrared light 305may be output toward the real world. The transmitted infrared light 305may be incident on the real object and may be partially reflected fromthe real object.

The display waveguide 350 and the eye tracking waveguide 360 may beincluded in a transparent member 370 (e.g., the first transparent member225 a and the second transparent member 225 b of FIG. 2 ). Thetransparent member 370 may be formed as, for example, a glass plate, aplastic plate, or a polymer, and may be transparently or translucentlyformed. The transparent member 370 may face an eye of a user. In thiscase, a distance between the transparent member 370 and the eye 301 maybe referred to as an “eye relief” 380.

The transparent member 370 may include the display waveguide 350 and theeye tracking waveguide 360. The transparent member 370 may include theinput optical member 330 and the output optical member 340. In addition,the transparent member 370 may include an eye tracking splitter 375 thatsplits the input light into several waveguides.

The display waveguide 350 is separated from the input optical member 330as shown in FIG. 3 , however, this is merely an example. The inputoptical member 330 may also be included in the display waveguide 350.

In addition, the output optical member 340 is separated from the eyetracking waveguide 360, as shown in FIG. 3 , however, this is merely anexample. The output optical member 340 may also be included in the eyetracking waveguide 360.

An optical waveguide (e.g., the display waveguide 350 and the eyetracking waveguide 360) may output a virtual object image by adjusting apath of visible light. Visible light and infrared light output from thedisplay 320 may be incident on the input optical member 330 through theprojection lens 325. Visible light among light incident on the inputoptical member 330 may be totally reflected through the displaywaveguide 350 to be guided to the output optical member 340. The visiblelight may be output from the output optical member 340 toward the eye301 of the user.

The wearable electronic device 300 may reflect or transmit the infraredlight output from the display 320 through the half mirror. In anembodiment, the wearable electronic device 300 may output the reflectedinfrared light 303 that is reflected by the half mirror (not shown)directly toward the eye 301 of the user, or may output the reflectedinfrared light 303 passing through the display waveguide 350 toward theeye 301 of the user. In an embodiment, the wearable electronic device300 may output the transmitted infrared light 305 passing through thehalf mirror toward the real object. A reflectivity and a transmittanceof the half mirror may be adjusted. For example, the half mirror mayhave a reflectivity of 30% (e.g., reflection toward eyes of a user) anda transmittance of 70% (e.g., output toward a real object) with respectto infrared light. However, the reflectivity and the transmittance aremerely examples and may be adjusted in various ratios.

In an embodiment, the wearable electronic device 300 may output thereflected infrared light 303 toward eyes of the user through the halfmirror and the infrared pixels included in the display 320. Thereflected infrared light 303 may be reflected from the eye 301 of theuser, and the eye tracking sensor 315 may detect the reflected light.The display 320 including the infrared pixels, and the half mirrorincluded in the display waveguide 350 may be used instead of a separateinfrared light source for detecting a real object. Since the separateinfrared light source is not used, the wearable electronic device 300may be lightened and power consumption may be reduced. In addition, thedisplay 320 including the infrared pixels may function as an auxiliarylight source to increase an image quality of a stereo camera (e.g., thefirst recognition camera 265 a and the second recognition camera 265 bof FIG. 2 ) in a low-illuminance environment and increase an accuracy ofdepth information.

Alternatively, the wearable electronic device 300 may output infraredlight through the display 320 and detect light reflected from the realobject through a stereo camera (e.g., the first recognition camera 265 aand the second recognition camera 265 b of FIG. 2 ). The wearableelectronic device 300 may estimate a distance to the real object basedon a detection result. For example, the wearable electronic device 300may measure a depth value or use a time of flight (ToF) scheme toestimate the distance to the real object.

The wearable electronic device 300 (e.g., the wearable electronic device200 of FIG. 2 ) may provide AR to a user. The wearable electronic device300 may provide an image representing the real world through thetransparent eye tracking waveguide 360, while transferring a virtualobject image output from the display 320 toward eyes of the user throughthe display waveguide 350.

The wearable electronic device 300 may include, but is not limited to,for example, a head-mounted display (HMD), a face-mounted display (FMD),or smart glasses (e.g., AR glasses) or a headset that provides extendedreality such as AR, VR, or mixed reality.

In an embodiment, the wearable electronic device 300 may output infraredlight using the display 320 including the infrared pixels. The wearableelectronic device 300 may track a gaze of a user, using the infraredlight output from the display 320. In addition, the wearable electronicdevice 300 may estimate a distance to a real object, using the infraredlight output from the display 320.

The electronic device according to an embodiment may be one of varioustypes of electronic devices. The electronic device may include, forexample, a portable communication device (e.g., a smart phone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance device.According to an embodiment of the disclosure, the electronic device isnot limited to those described above.

It should be understood that an embodiment of the disclosure and theterms used therein are not intended to limit the technological featuresset forth herein to particular embodiments and include various changes,equivalents, or replacements for a corresponding embodiment. Inconnection with the description of the drawings, like reference numeralsmay be used for similar or related components. It is to be understoodthat a singular form of a noun corresponding to an item may include oneor more of the items, unless the relevant context clearly indicatesotherwise. As used herein, “A or B,” “at least one of A and B,” “atleast one of A or B,” “A, B or C,” “at least one of A, B and C,” and “atleast one of A, B, or C,” each of which may include any one of the itemslisted together in the corresponding one of the phrases, or all possiblecombinations thereof. Terms such as “1^(st)”, “2^(nd)” or “first” or“second” may simply be used to distinguish the component from othercomponents in question, and do not limit the components in other aspects(e.g., importance or order). It is to be understood that if an element(e.g., a first element) is referred to, with or without the term“operatively” or “communicatively,” as “coupled with,” “coupled to,”“connected with,” or “connected to” another element (e.g., a secondelement), it denotes that the element may be coupled with the otherelement directly (e.g., by wire), wirelessly, or via a third element.

As used in connection with an embodiment of the disclosure, the term“module” may include a unit implemented in hardware, software, orfirmware, and may interchangeably be used with other terms, for example,“logic,” “logic block,” “part,” or “circuitry.” A module may be a singleintegral component, or a minimum unit or part thereof, adapted toperform one or more functions. For example, according to an embodiment,the module may be implemented in a form of an application-specificintegrated circuit (ASIC).

An embodiment as set forth herein may be implemented as software (e.g.,the program 140) including one or more instructions that are stored in astorage medium (e.g., an internal memory 136 or an external memory 138)that is readable by a machine (e.g., the electronic device 101). Forexample, a processor (e.g., the processor 120) of the machine (e.g., theelectronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium and execute it. This allowsthe machine to be operated to perform at least one function according tothe at least one instruction invoked. The one or more instructions mayinclude code generated by a compiler or code executable by aninterpreter. The machine-readable storage medium may be provided in theform of a non-transitory storage medium. Here, the term “non-transitory”simply denotes that the storage medium is a tangible device, and doesnot include a signal (e.g., an electromagnetic wave), but this term doesnot differentiate between where data is semi-permanently stored in thestorage medium and where the data is temporarily stored in the storagemedium.

According to an embodiment, a method according to an embodiment of thedisclosure may be included and provided in a computer program product.The computer program product may be traded as a product between a sellerand a buyer. The computer program product may be distributed in the formof a machine-readable storage medium (e.g., compact disc read-onlymemory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smartphones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asa memory of the manufacturer's server, a server of the applicationstore, or a relay server.

According to an embodiment, each component (e.g., a module or a program)of the above-described components may include a single entity ormultiple entities, and some of the multiple entities may be separatelydisposed in different components. According to an embodiment, one ormore of the above-described components or operations may be omitted, orone or more other components or operations may be added. Alternativelyor additionally, a plurality of components (e.g., modules or programs)may be integrated into a single component. In such a case, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to an embodiment, operations performed by themodule, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

According to an embodiment, an electronic device may register anauthentication sequence in a physical space and perform userauthentication based on whether authentication objects are selected inthe physical space according to the registered authentication sequence.The authentication sequence may be a sequence in which authenticationobjects are selected for user authentication. The authenticationsequence may be registered for an authority related to at least one of aphysical space or an electronic device. The authentication sequence maybe registered for at least one of information (e.g., informationindicating a virtual background, a virtual space that augments aphysical space) mapped to and stored in a physical space, a deviceplaced in the physical space, and a device connected to an electronicdevice in the physical space. For example, when the electronic devicedetermines that user authentication for the physical space issuccessful, the electronic device may allow the corresponding usercontrol authority of a device placed in the corresponding physicalspace. As another example, based on successful user authentication, theelectronic device may grant access to information (e.g., content mappedto the corresponding space by the user or content set to be viewed onlyin the corresponding space) available in the corresponding physicalspace to the user.

Hereinafter, FIGS. 4 and 5 describe a user authentication operationbased on matching between a registered authentication sequence and aninput authentication sequence, and FIG. 11 describes the registration ofan authentication sequence.

FIGS. 4 and 5 are flowcharts illustrating a user authentication methodaccording to various embodiments of the disclosure.

In the following embodiments, operations may be performed sequentially,but not necessarily sequentially. For example, the order of theoperations may change, and at least two of the operations may beperformed in parallel.

FIG. 4 may show an operation flow when some of the authenticationobjects are unavailable.

Referring to FIG. 4 , in operation 410, an electronic device (e.g., theelectronic device 101 of FIG. 1 ) may obtain current spatial informationon a physical space where a user is located using a sensor (e.g., thesensor module 176 of FIG. 1 ). The current spatial information mayinclude a geographic location of a physical space where the user islocated, a size of the space, a shape of the space, a position of aphysical object placed in the space, a size of the physical object, anda shape of the physical object. For example, the current spatialinformation may be information on a portion of a physical spacebelonging to a scene within a sensing range (e.g., a field of view) of asensor based on a direction the electronic device is currently facingfrom its current position. The sensor may collect information to provideaugmented reality. For example, the sensor may include at least one of acamera, an infrared sensor, a depth sensor (e.g., a lidar sensor, aradar sensor, or a stereo camera), a gyro sensor, an accelerationsensor, or a geomagnetic sensor. The electronic device may detect apartial space within a field of view of the user and/or a field of viewof the sensor and physical objects placed in the partial space based ona direction in which the user's head and/or the electronic device isfacing in the physical space. The physical objects are described belowwith reference to FIG. 6 .

When all authentication objects included in the authentication sequencethat is registered in the current spatial information are available, theelectronic device may receive an input for selecting an object in thespace from the user. The authentication object may be an objectregistered for the aforementioned authentication and may be a physicalobject. The electronic device may generate and output a virtual objectcorresponding to a physical object in the space. The electronic devicemay overlay the virtual object on the physical object and output thevirtual object through an AR display. A virtual object corresponding toa physical object may have the same or similar shape as thecorresponding physical object, but is not limited thereto. Forreference, embodiments of the disclosure mainly describe an example ofreplacing a physical object with a virtual object and outputting thevirtual object, but the disclosure is not limited thereto. Instead ofoutputting a virtual object for a physical object, the electronic devicemay set a manipulation area in a space occupied by the physical objector a position corresponding to the physical object. The manipulationarea may be set to have the same shape and volume as the space occupiedby the physical object, but is not limited thereto. The electronicdevice may determine that the physical object corresponding to theselected manipulation area is selected by the user, based on detecting auser input for selecting the manipulation area.

In addition, a virtual object may include a physical object actuallydisposed in a physical space as well as a virtual object temporarilygenerated for authentication, which will be described later.

In operation 420, the electronic device may generate a greater number ofvirtual objects than the number of unavailable authentication objects,based on at least one authentication object among a plurality ofauthentication objects being unavailable in the current spatialinformation. The electronic device may generate a virtual object as atemporary object to replace an unavailable authentication object. Theelectronic device may prevent a third party using the electronic devicefrom inferring an authentication object other than a legitimate user bygenerating and outputting a greater number of temporary objects than thenumber of unavailable authentication objects.

The electronic device may determine whether the correspondingauthentication object is unavailable based on at least one of whether anobject is detected, whether a state has changed, whether a position haschanged, or whether a shape has changed. For example, the electronicdevice may determine at least one of an object out of a field of view ofa sensor, an object out of a predetermined distance range from theelectronic device, an object deformed from a registered state, an objectmoved from a registered position, or an undetected object among aplurality of authentication objects to be an unavailable authenticationobject. An object out of a field of view may be an object that is notdetected from the sensing data of the sensor and may be an object thatis out of the field of view or separated from the electronic device by adistance that is difficult to identify. An object out of a predetermineddistance range may be an object separated from the electronic device bya threshold distance or more, or less than a minimum distance. Thepredetermined distance range may be a range representing a distance atwhich a selection input by a user is possible, and may be set to excludeobjects that are too close or too far. An object deformed from aregistered state may be an object in which a current state and aregistered state are different. For example, when an object was a potwith a plant in the registered state but is a pot without the plant inthe current state, the electronic device may determine that the state ofthe object has been deformed. An object moved from a registered positionmay be an object in which a registered position and a currently detectedposition are different.

In operation 430, the electronic device may output the generated virtualobjects to a plurality of positions including a physical position of anunavailable authentication object. The electronic device may generatethe virtual objects using a display providing augmented reality. Thedisplay may include at least one of an optical see-through display or avideo see-through display. The optical see-through display may transmitlight reflected from a physical object in the real world to the eyes ofthe user and provide image information generated based on the virtualobject. In FIGS. 2 and 3 , an example configuration of an opticalsee-through display has been described. The video see-through displaymay provide image information generated based on a physical object and avirtual object captured by a camera.

The electronic device may prevent speculation of an authenticationobject by a third party and ensure security by outputting a temporarilygenerated virtual object to another position as well as the position ofthe unavailable authentication object. The other position may include aposition of another physical object within a physical space. The otherposition may include a randomly selected position within the physicalspace.

In operation 440, the electronic device may determine thatauthentication is successful based on a user input that selects thegenerated virtual object and an available authentication object as aregistered sequence. For example, the electronic device may detect auser input that sequentially selects one or more objects from among thevirtual objects output in operation 430. The user input will bedescribed below with reference to FIG. 7 . The electronic device maydetermine a sequence for selecting a virtual object and whether theselected virtual object matches a registered authentication sequence anda registered authentication object. The electronic device may determinethat authentication is successful based on a sequence input by the usermatching an authentication sequence. Based on the authentication beingsuccessful, the electronic device may allow access to at least one ofinformation or a device related to at least one of the physical space orthe electronic device.

The electronic device according to an embodiment may provide an ARenvironment to a user wearing a corresponding device (e.g., an ARdevice). The electronic device may augment a virtual space and a realspace by naturally arranging a virtual object or virtual space in a realphysical environment (e.g., a physical space). The electronic device mayprovide a user with a function to call and efficiently control thevirtual space for various purposes such as work, leisure, andappreciation. The electronic device may provide a unique virtual spaceto the user and guarantee the privacy and security of the user bypreventing access by a third party through the above-describedauthentication operation. In addition, the electronic device maydetermine whether control authority for the electronic device itself ora device connected to the electronic device is allowed through theabove-described authentication operation.

The electronic device may identify an order (e.g., a sequence) forselecting a plurality of objects arranged in a space (e.g., a virtualspace) by recognizing a user input including a gaze or a gesture (e.g.,hand gesture) of the user, which will be described later. The electronicdevice may allow the control authority based on the sequential selectionof registered authentication objects according to a predefinedauthentication sequence. Based on the detection of the unavailableauthentication object as described above, the electronic device mayprovide a stable authentication operation to the user and preventleakage of an authentication object to a third party by outputting agreater number of temporary virtual objects than the unavailable number(e.g., the number of unavailable authentication objects). Therefore, theelectronic device may provide an authentication operation that is robustagainst changes including non-detection, deformation, or obscuration ofa physical object included in previous spatial information.

FIG. 5 may show an operation flow of an authentication operation using avirtual object.

An electronic device (e.g., the electronic device 101 of FIG. 1 )according to an embodiment may initiate an authentication operation forcontrolling and/or sharing a virtual space in a real space (e.g., aphysical space).

Referring to FIG. 5 , in operation 511, the electronic device may obtaincurrent spatial information. The electronic device may obtainthree-dimensional (3D) information about an actual physical space byrecognizing the real space. The obtaining of the current spatialinformation will be further described below with reference to FIG. 6 .

In operation 512, the electronic device may determine whether there is apreviously registered authentication space in a space where a user islocated. When there is no previously registered authentication space, inoperation 580, the electronic device may determine whether to register anew authentication sequence. In operation 590, the electronic device mayinitiate authentication registration based on the determination toregister a new authentication sequence based on a user input. Theauthentication registration will be further described below withreference to FIG. 11 .

In operation 513, the electronic device may determine whether previousspatial information and current spatial information do not match. Theelectronic device may receive an authentication sequence from the userin operation 541 based on the previous spatial information matching thecurrent spatial information.

In operation 514, the electronic device may determine whether an errorbetween the previous spatial information and the current spatialinformation is less than a threshold error. For example, the electronicdevice may calculate an error (e.g., a spatial error) between spacesbased on an object included in the previous spatial information and anobject included in the current spatial information. The spatial errormay be a ratio of the number of objects included in one piece of spatialinformation to the number of objects not detected in another piece ofspatial information, but is not limited thereto. The electronic devicemay stop the authentication operation when an error between the spatialinformation is equal to or greater than the threshold error. In thiscase, the electronic device may proceed with the registration of a newauthentication sequence or terminate the authentication operation. Theelectronic device may continue the authentication operation when theerror between the spatial information is less than the threshold error.

In operation 520, the electronic device may generate a virtual object.For example, as described in relation to operation 420 of FIG. 4 , theelectronic device may generate a temporary virtual object to replace anunavailable authentication object. The electronic device may generate agreater number of virtual objects than the unavailable number in orderto prevent leakage to a third party.

In operation 530, the electronic device may output a virtual object. Forexample, as described in relation to operation 430 of FIG. 4 , theelectronic device may output a virtual object corresponding to aphysical object currently existing in the physical space.

In operation 541, the electronic device may receive an input of anauthentication sequence from the user. The authentication sequence inputmay represent a user input for sequentially selecting a virtual object.For example, the electronic device may identify an authenticationsequence according to a gaze of the user or an order of instruction.

For reference, when an authentication object is obscured due to a changein a position or a gaze direction of the user, the electronic device mayprovide a guide for the movement of the user position to induce matchingof the user's gaze direction. For example, based on a current gazedirection of the user being different from a gaze direction registeredwhen the authentication sequence is registered, the electronic devicemay output information guiding at least one of a position or posture inwhich an error between the current gaze direction and the registeredgaze direction at the time of registration is less than a thresholderror. The registered gaze direction may represent a gaze direction(e.g., a direction the electronic device faces or a direction eyes ofthe user gaze at) of the user in the previous spatial information (e.g.,spatial information upon registration). The current gaze direction mayrepresent a direction the user gazes at or a direction the electronicdevice faces in the current space. The electronic device may outputinformation guiding at least one of a position or a direction in whichan error between the current gaze direction and the registered gazedirection is reduced, to the user. Accordingly, the electronic devicemay enhance the convenience of inducing the user to input theauthentication sequence in the same position and direction as at thetime of registration.

The electronic device may determine the gaze direction based on at leastone of sensing or head tracking of an eye tracking camera. For example,the electronic device may determine a direction in which the eyes of theuser are gazing based on sensing data of the eye tracking camera. Asanother example, the electronic device may determine a heading directionof the electronic device based on head tracking and determine a gazedirection based on the heading direction of the electronic device. Theheading direction of the electronic device may be, for example, adirection perpendicular to a corresponding one surface from the centerof one surface (e.g., a front surface) of the electronic device, and theheading direction of the electronic device may be determined as the gazedirection. However, the determination of the gaze direction is notlimited to the above-described example.

In addition, the electronic device may output guidance information forinducing a change from a current environment to a registeredenvironment, based on the current environment in which authentication isbeing attempted being different from the registered environment when theauthentication sequence is registered. For example, the electronicdevice may output information guiding an action to decrease a differencein illuminance, based on a difference between current illuminance andprevious illuminance at the time of registration. For example, when theilluminance at the time of registration is higher than the currentilluminance, the electronic device may guide the user to activatelighting in the current physical space. Accordingly, the electronicdevice may provide the convenience of inducing the user to performauthentication in the same environment as at the time of registration.

In operation 542, the electronic device may determine whether the inputauthentication sequence matches the registered authentication sequence.The electronic device may determine whether the user has selected eachregistered authentication object in the input authentication sequenceaccording to the registered authentication sequence.

In operation 543, the electronic device may determine thatauthentication is successful when the authentication matches. Theelectronic device may determine that authentication is successful basedon a selection sequence of a user for an authentication object providedto augmented reality through a display for current spatial informationmatching a pre-registered authentication pattern. The electronic devicemay store and terminate the authentication result. However, when theauthentications do not match, the electronic device may terminate theauthentication when the authentications do not match a predeterminednumber of times or more.

As described above, the electronic device according to an embodiment mayprovide an authentication operation using an object in a 3D space of avirtual space augmented in a real environment to a user wearing thecorresponding device. The electronic device is intuitive and may provideinformation (e.g., an authentication sequence) that may be easilygenerated only by legitimate users with high security. Based on theabove-described authentication operation, the electronic device maydetermine whether to grant control authority of a virtual space of auser, whether to allow control of the virtual space, or whetherauthorization has been authenticated in a case where a virtual space isshared such as when enjoying a space with other users or conductingcollaborative work. The electronic device may analyze an actual spaceand compare the actual space with a previously registered space, toprovide space authentication tailored to the user even when the space tobe authenticated is partially different from the stored space.

FIG. 6 is a diagram illustrating an example of obtaining current spatialinformation according to an embodiment of the disclosure.

Referring to FIG. 6 , an electronic device 600 (e.g., the electronicdevice 101 of FIG. 1 ) according to an embodiment may analyze a physicalspace as 3D information by using various input signals (e.g., sensingdata of an RGB camera, infrared sensor, depth sensor, or stereo camera)of sensors. For example, the electronic device 600 may analyze at leastone of a shape, size, or position of a physical space, or a shape, size,or position of a physical object.

The electronic device 600 may detect an object captured in a scenecorresponding to a field of view 691 of a camera using sensing data(e.g., a captured image) of the camera. The electronic device 600 maydetermine a label (e.g., information indicating the classification of anobject, including a value indicating a chair, monitor, or plant) of aphysical object and an area (e.g., a bounding box) occupied by thephysical object within a two-dimensional (2D) scene from a 2D sceneimage of the camera. Accordingly, the electronic device 600 may obtain2D scene information 611 from a position where a user 690 is looking. Inaddition, the electronic device 600 may calculate a position of theelectronic device 600 in a physical space based on sensing data of thecamera.

The electronic device 600 may obtain position information of the user690 and depth information 612 of a real space in a viewing directionusing sensing data (e.g., depth data) of a depth sensor. The depthinformation 612 may be information indicating a distance from the depthsensor to each point and may be expressed in the shape of a depth map.The electronic device 600 may analyze a distance of each pixel unit froma 3D position viewed by the user 690.

The electronic device 600 may obtain information 613 including a 3Dpoint cloud and a mesh using various sensing data. The electronic device600 may obtain a plane, a mesh, or a 3D coordinate point clusterincluded in a space by analyzing the physical space. The electronicdevice 600 may obtain a 3D point cloud representing physical objectsbased on the information obtained as described above.

The electronic device 600 may analyze the physical space and obtaininformation 614, 615, 616 including at least one of 3D positioncoordinates, a 3D shape, or 3D size (e.g., a 3D bounding box) of thephysical objects arranged in the physical space. Accordingly, theelectronic device 600 may obtain object detection information andsemantic segmentation information of the 3D space.

As described above, the electronic device 600 may obtain and store 3Dinformation (e.g., spatial information) on a physical object and aphysical space. The electronic device 600 may store 3D positioninformation in a space of the user 690 together with the spatialinformation. The electronic device 600 may store 3D spatial information650 upon registration.

When an authentication operation is performed, the electronic device 600may compare current spatial information with previous spatialinformation. The current spatial information and previous spatialinformation may be stored as the 3D spatial information 650 as shown inFIG. 6 . The electronic device 600 may calculate similarity informationbased on the comparison between the current spatial information andprevious spatial information. For example, the electronic device 600 mayalso calculate a similarity between the stored previous spatialinformation and current spatial information as a ratio between 0% and100%, but the example is not limited thereto. The electronic device 600may correct and compare position information of the user 690 of theprevious spatial information with position information of the user 690of the current spatial information. As described above, when theposition and gaze direction 692 of the user 690 in the previous spatialinformation and the current spatial information are different, theelectronic device 600 may provide guidance to the user 690 to take thesame or similar position and gaze direction 692 as the previous spatialinformation. The electronic device 600 may provide an authenticationoperation that is robust against various position changes of the user690 in the same real space.

FIG. 7 is a diagram illustrating an operation of selecting anauthentication object according to an embodiment of the disclosure.

Referring to FIG. 7 , an electronic device 700 (e.g., the electronicdevice 101 of FIG. 1 ) according to an embodiment may identify at leastone of a gaze direction 792 or a gesture of a user 790 using a sensor.The electronic device 700 may detect the selection of virtual objectswithin a field of view 791 of the sensor. For example, the electronicdevice 700 may determine that a corresponding virtual object is selectedbased on detecting an action 720 of gazing at the virtual object for apredetermined time or more through eye gaze tracking technology. Theelectronic device 700 may recognize a gesture pointing to the virtualobject through hand tracking technology. The electronic device 700 maydetermine that the corresponding virtual object is selected when adirection 710 in which a tracked hand is pointing points to the virtualobject for a predetermined time or more, or the hand of the user 790contacts or enters an area occupied by the virtual object in the virtualspace.

Accordingly, the electronic device may identify an authenticationsequence input by the user based on an order in which the eyes of theuser gaze or the hand of the user points to the authentication object.

FIG. 8 is a diagram illustrating types of authentication objectsaccording to an embodiment of the disclosure.

Referring to FIG. 8 , an electronic device 800 (e.g., the electronicdevice 101 of FIG. 1 ) according to an embodiment may generate a virtualobject corresponding to a real physical object or a temporary object. Asdescribed above, the electronic device 800 may generate a virtual objectbased on a result of analyzing a physical space in reality. In an ARenvironment, the virtual object may have a 3D shape similar to a realenvironment or a 3D shape that a user 890 may easily distinguish andperceive with the eyes. The electronic device 800 may generate a virtualobject that is selectable according to an instruction based on a gaze orgesture of the user 890.

For example, the electronic device 800 may generate the virtual objectusing a 2D object 810 in 2D scene information of a position (e.g., agaze direction and position of the user 890) where the user 890 islooking within a sensing range of a sensor. The electronic device 800may generate a 2D image object as a virtual object in a virtual spaceand may provide a reduced or enlarged 2D image object. A size of the 2Dimage object may vary according to a preference of the user 890.

The electronic device 800 may generate a virtual object 820corresponding to a virtual plane at the same position as a real physicalenvironment. For example, the electronic device 800 may generate avirtual object having a shape corresponding to a floor or wall which hasa surface shape in the real environment.

The electronic device 800 may provide a 3D virtual object 830 at thesame position as the real environment. In FIG. 8 , the 3D virtual object830 is shown in the form of a 3D bounding box displayed at a positionwhere a physical object in a real environment is arranged. For example,the electronic device 800 may arrange a virtual object at 3D positioncoordinates of the physical object in the real environment.

The electronic device 800 may provide a 3D virtual object 840 having thesame position and size as the real environment. In FIG. 8 , the 3Dvirtual object 840 is shown as a virtual object having the same shapeand size as a physical object, determined based on segmentation of a 3Dscene. However, the embodiment is not limited thereto. As describedabove, a manipulation area smaller than that of the physical object maybe set for a virtual object based on a physical object.

The electronic device 800 may provide virtual objects of various objecttypes shown in FIG. 8 in an authentication operation. The electronicdevice 800 may output virtual objects including authentication objects,temporary objects, and objects corresponding to other physical objectsnot registered for authentication at a plurality of positions.

FIG. 9 is a diagram illustrating an operation of selecting anauthentication object according to an authentication sequence accordingto an embodiment of the disclosure.

Referring to FIG. 9 , an electronic device (e.g., the electronic device101 of FIG. 1 ) according to an embodiment may detect a user input forselecting one or more virtual objects among a plurality of virtualobjects 911, 912, 913, and 914 arranged in a space. As described abovewith reference to FIG. 7 , a user input by a user 990 may be an inputindicating a virtual object. The electronic device 900 may reproduce avirtual space by combining the virtual space with a physical space. Inaddition, the electronic device 900 may reproduce the plurality ofvirtual objects 911, 912, 913, and 914 by combining them with a virtualspace and/or a physical space. For example, the electronic device 900may arrange and output the plurality of virtual objects 911, 912, 913,and 914 at corresponding positions. The electronic device 900 may applya motion or an animation effect to the virtual space and the pluralityof virtual objects 911, 912, 913, and 914.

The electronic device 900 may detect a user input for sequentiallyselecting the plurality of virtual objects 911, 912, 913, and 914arranged in the virtual space for authentication. The electronic device900 may determine whether an order of selecting the plurality of virtualobjects 911, 912, 913, and 914 matches a registered authenticationsequence. Hereinafter, an example in which some of the authenticationobjects included in the registered authentication sequence areunavailable is described with reference to FIG. 10 .

FIG. 10 is a diagram illustrating the generation of a virtual objectwhen some authentication objects are unavailable according to anembodiment of the disclosure.

Referring to FIG. 10 , an electronic device (e.g., the electronic device101 of FIG. 1 ) according to an embodiment may perform authenticationeven when some authentication objects are unavailable. Examples of whensome authentication objects are unavailable may include, when a physicalobject is gone, when the physical object is deformed and changed, whenthe physical object is not visible from a visibility of a user, or whena current space is different from a previously stored space. Theelectronic device may automatically reconstruct a space or provide avirtual object that replaces a real object to a user. The electronicdevice may also recommend registration of a new authentication sequence,which will be described later.

For example, the electronic device may generate virtual objects in ashape different from an unavailable authentication object 1010. Theelectronic device may generate the virtual objects in at least one of adefault shape predetermined by a user or a shape of a physical objectdetected within a sensing range of a sensor. As shown in FIG. 10 , theelectronic device may output virtual objects 1020 having a monkey shapeas a predetermined default shape to a virtual space. The electronicdevice may output virtual objects 1030 having lamp shapes as shapes ofphysical objects detected in the physical space to the virtual space.The electronic device may prevent security leakage to a third party byarranging the virtual objects 1020 at positions of registeredauthentication objects as well as at positions of other physical objectsin the physical space. The electronic device may generate at least onevirtual object among virtual objects in the same shape as anothervirtual object. For reference, an example in which all of the virtualobjects 1020 and 1030 have the same shape is illustrated.

In addition, the electronic device may generate at least one virtualobject as an object having at least one of a ratio or a size differentfrom that of another virtual object while having the same shape as theother virtual object. According to an embodiment of the disclosure, ashape may represent an appearance independent of size or proportion.FIG. 10 shows an example in which the virtual objects 1020 and 1030having different ratios and sizes of a monkey shape and a lamp shape areoutput.

However, the example is not limited to the foregoing, and the electronicdevice may randomly select the shapes of virtual objects. The electronicdevice may generate at least one virtual object among virtual objects ina shape different from another virtual object. For example, a virtualobject 1041 having a different shape from the rest may be arranged inplace of the unavailable authentication object 1010. Also, a virtualobject 1042 having the same shape as the unavailable authenticationobject 1010 may be arranged.

The electronic device may arrange at least one virtual object amongvirtual objects at the same position as a currently availableauthentication object. The electronic device may determine at least somepositions among a plurality of positions where the virtual objects areoutput as positions of a physical object detected in a field of view ofa sensor in a current place of the electronic device. Accordingly, theplurality of positions where the virtual objects are output may includenot only a position of the unavailable authentication object 1010 and aposition where the currently available authentication object isarranged, but also a position of a physical object not included in theregistered authentication sequence. The electronic device may overlayand output an object arranged at a position of a currently availableauthentication object among virtual objects on the currently availableauthentication object. The electronic device may also overlay and outputthe virtual object on another physical object existing in the currentspace.

When some of the authentication objects are unavailable, the electronicdevice according to an embodiment may provide a virtual object having ashape different from a previously registered authentication object tosecure a registered authentication sequence. In this case, a virtualobject having the same shape may be overlaid and output at a position ofa physical object included in the registered authentication sequence orat a position of another physical object not included in theauthentication sequence. By outputting a virtual object to a position ofthe unavailable authentication object 1010, a position of the availableauthentication object, and a position of another physical object, theelectronic device may provide authentication services to legitimateusers and prevent security leakage to third parties.

In addition, the electronic device may limit the output of virtualobjects at a position of a physical object that is out of at least oneof a predetermined size range or a distance range predetermined by theelectronic device from among physical objects detected in the field ofview of the sensor. The predetermined size range may be a range betweena minimum size and a maximum size and may be determined based on atleast one of a user's manipulation convenience or object visibility. Theminimum size may be determined as the size of the unavailableauthentication object 1010, for example. Positions of physical objectsthat are too large or too small may be excluded. When there is abookshelf in the current space, inconvenience in manipulation that maybe caused when virtual objects are displayed on all the books may beprevented. The predetermined distance range may be a range between aminimum distance and a maximum distance and may be determined based onat least one of a user's manipulation convenience or object visibility.Positions of physical objects that are too close or too far may beexcluded. Inconvenience in manipulation that may be caused when avirtual object is displayed at a position that is too far away (that is,a situation where it is difficult for a user to select a virtual objectdisplayed at a position that is too far away) may be prevented.

As described above, the electronic device according to an embodiment mayoutput a virtual object having a default shape or a virtual objectobtained by copying a shape identical to a physical object within afield of view of a sensor. The electronic device may generate a mixtureof virtual objects having a shape unrelated to the registeredauthentication sequence or a shape related to the authenticationsequence. The electronic device may enhance security by replacing andoutputting the unavailable authentication object 1010 with a pluralityof virtual objects having the above-described shapes.

FIG. 11 is a flowchart illustrating an operation of registering a newauthentication sequence according to an embodiment of the disclosure.

Referring to FIG. 11 , an electronic device (e.g., the electronic device101 of FIG. 1 ) according to an embodiment may recommend an object to beused for authentication to a user based on current spatial informationwhen an authentication sequence is registered. The electronic device mayinitiate an operation of registering and/or updating a newauthentication sequence from operation 590 described above withreference to FIG. 5 . For example, the electronic device may initiate anoperation for registering a new authentication sequence based on theuser authentication being successful as described above with referenceto FIGS. 4 and 5 . However, the example is not limited thereto, and theelectronic device may initiate an operation for registering a newauthentication sequence based on determining a legitimate user for aspace subject to user authentication through various methods (e.g.,authentication based on a personal identification number (PIN) input,authentication based on biometric information including fingerprint andface). In addition, along with the above-described successful userauthentication, the electronic device may perform an operation forupdating an authentication sequence based on detecting a changed portionby comparing pre-registered previous spatial information with currentspatial information.

In operation 1110, the electronic device may obtain current spatialinformation. The electronic device may obtain 3D information on a sizeand shape of a current space and a size and shape of a physical objectwithin the current space. Since obtaining of the spatial information hasbeen described above, a further description is omitted.

In operation 1120, the electronic device may detect a physical object inthe current space. The electronic device may generate a virtual objectbased on the detected physical object. The electronic device may overlayand output a 3D virtual object individually corresponding to each of aplurality of the detected physical objects. The electronic device mayreproduce the virtual object by arranging the virtual object at acorresponding position and applying a motion effect.

In operation 1130, the electronic device may determine whether thenumber of physical objects is greater than or equal to a thresholdnumber. For example, the electronic device may determine whether virtualobjects are generated for N or more physical objects. In this example, Nmay be an integer greater than or equal to 2. The electronic device mayretry detection of the physical object in operation 1120 based on thenumber of the generated virtual objects being less than or equal to thethreshold number.

In operation 1140, the electronic device may recommend a candidateobject. The electronic device may recommend M candidate objects amongthe generated N virtual objects. M may be an integer greater than orequal to 1 and less than or equal to N. When previous spatialinformation exists, the electronic device may determine a candidateobject based on a comparison between the previous spatial informationand the current spatial information. The electronic device may recommenda candidate object using a changed portion of the current spatialinformation from the previous spatial information, based on a spatialerror between the previous spatial information and the current spatialinformation being less than a threshold error. For example, theelectronic device may present one or more recommended objects to be usedfor authentication based on the changed portion of the current spatialinformation from the registered previous spatial information. Theelectronic device may determine a physical object corresponding to thechanged portion to be the recommended object. Accordingly, theelectronic device may induce the changed portion to be included in theauthentication sequence.

In addition, the electronic device may recommend a target device as acandidate object based on registering an authentication sequence foraccessing the target device to be controlled. Accordingly, theelectronic device may register an authentication sequence intuitivelymapped to a control target.

Based on registering the authentication sequence, the electronic devicemay recommend, as a candidate object for the authentication sequence, aphysical object belonging to an object type determined based on apreference of a user among spatial information corresponding to aphysical space where the user is positioned. For example, when the useris a reader and a book is an object type preferred by the user, theelectronic device may preferentially recommend a virtual objectcorresponding to the book as a candidate object. The electronic devicemay intuitively guide the user that the object is the recommended objectby reproducing the M candidate objects by applying at least one of amotion or an animation effect to the recommended M candidate objects.

In operation 1150, the electronic device may receive an input of anauthentication sequence to be registered. The electronic device maystore an order of virtual objects sequentially selected by the user. Asdescribed above, the electronic device may determine whether a virtualobject has been selected based on at least one of eye tracking or handtracking.

In operation 1160, the electronic device may determine whether toconfirm the authentication sequence to be registered. The electronicdevice may determine whether the same authentication sequence isrepeatedly inputted by the user T or more times. T may be an integergreater than or equal to 1.

In operation 1170, the electronic device may store current spatialinformation based on the generating of enough virtual objects toregister a new authentication sequence.

In operation 1180, the electronic device may store the current spatialinformation, the authentication object, and the authentication sequencebased on confirming that the user inputs the same authenticationsequence. The electronic device may group and store the current spatialinformation, the authentication object, and the authentication sequence.The electronic device may store a selected object as an authenticationobject and register a selected order as an authentication sequence.Accordingly, the electronic device may update the authentication objectand the authentication sequence based on objects selected by the userfrom among candidate objects including one or more recommended objectand a sequence in which the objects are selected. For reference, anauthentication sequence according to an embodiment of the disclosure mayalso be referred to as an authentication pattern.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic device, comprising: a displaycircuit configured to provide augmented reality; a sensor configured tocollect information for providing the augmented reality; a memoryconfigured to store computer-executable instructions; and a processorconfigured to execute the instructions by accessing the memory, whereinwhen the instructions are executed by the processor, the processor isconfigured to: obtain current spatial information on a physical spacewhere a user is positioned using the sensor, generate a greater numberof virtual objects than the number of unavailable authenticationobjects, based on at least one authentication object among a pluralityof authentication objects being unavailable in the current spatialinformation, output the generated virtual objects to a plurality ofpositions comprising a physical position of the unavailableauthentication object, and determine that authentication is successfulbased on a user input that selects a generated virtual object and anavailable authentication object as a registered sequence.
 2. Theelectronic device of claim 1, wherein, when the instructions areexecuted by the processor, the processor is further configured to:generate the virtual objects in a shape different from the unavailableauthentication object.
 3. The electronic device of claim 1, wherein,when the instructions are executed by the processor, the processor isfurther configured to: generate the virtual objects in at least one of adefault shape predetermined by a user or a shape of a physical objectdetected within a sensing range of the sensor.
 4. The electronic deviceof claim 1, wherein, when the instructions are executed by theprocessor, the processor is further configured to: generate at least onevirtual object among the virtual objects in the same shape as anothervirtual object.
 5. The electronic device of claim 4, wherein, when theinstructions are executed by the processor, the processor is furtherconfigured to: generate the at least one virtual object as an objecthaving at least one of a ratio or a size different from that of theother virtual object while having the same shape as the other virtualobject.
 6. The electronic device claim 1, wherein, when the instructionsare executed by the processor, the processor is further configured to:randomly select shapes of the virtual objects, and generate at least onevirtual object among the virtual objects in a shape different fromanother virtual object.
 7. The electronic device of claim 1, wherein,when the instructions are executed by the processor, the processor isfurther configured to: arrange at least one virtual object among thevirtual objects at the same position as a currently availableauthentication object.
 8. The electronic device of claim 1, wherein,when the instructions are executed by the processor, the processor isfurther configured to: overlay and output an object arranged at aposition of a currently available authentication object among thevirtual objects on the currently available authentication object.
 9. Theelectronic device of claim 1, wherein, when the instructions areexecuted by the processor, the processor is further configured to:determine at least some positions among a plurality of positions wherethe virtual objects are output as positions of a physical objectdetected in a field of view of the sensor in a current place of theelectronic device.
 10. The electronic device of claim 1, wherein, whenthe instructions are executed by the processor, the processor is furtherconfigured to: limit output of the virtual objects at a position of aphysical object that is out of at least one of a predetermined sizerange or a distance range predetermined by the electronic device fromamong physical objects detected in a field of view of the sensor. 11.The electronic device of claim 1, wherein, when the instructions areexecuted by the processor, the processor is further configured to:determine at least one of an object out of a field of view of thesensor, an object out of a predetermined distance range from theelectronic device, an object deformed from a registered state, an objectmoved from a registered position, or an undetected object among theplurality of authentication objects as the unavailable authenticationobject.
 12. The electronic device of claim 1, wherein, when theinstructions are executed by the processor, the processor is furtherconfigured to: based on a current gaze direction of the user beingdifferent from a gaze direction registered when an authenticationsequence is registered, output information guiding at least one of aposition or a posture in which an error between the current gazedirection and the registered gaze direction at a time of registration isless than a threshold error.
 13. The electronic device of claim 1,wherein, when the instructions are executed by the processor, theprocessor is further configured to: output guidance information forinducing a change from a current environment to a registeredenvironment, based on the current environment in which authentication isbeing attempted being different from the registered environment when anauthentication sequence is registered.
 14. The electronic device ofclaim 1, wherein, when the instructions are executed by the processor,the processor is further configured to: perform an operation forupdating an authentication sequence based on detecting a changed portionby comparing pre-registered previous spatial information with thecurrent spatial information.
 15. The electronic device of claim 14,wherein, when the instructions are executed by the processor, theprocessor is further configured to: present one or more recommendedobjects to be used for authentication based on the changed portion ofthe current spatial information from the pre-registered previous spatialinformation, and update an authentication object and an authenticationsequence based on objects selected by the user from among candidateobjects comprising the one or more recommended objects and a sequence inwhich the objects are selected.
 16. The electronic device of claim 1,wherein, when the instructions are executed by the processor, theprocessor is further configured to: recommend a target device as acandidate object based on registering an authentication sequence foraccessing the target device to be controlled.
 17. The electronic deviceof claim 16, wherein, when the instructions are executed by theprocessor, the processor is further configured to: based on registeringthe authentication sequence, recommend, as a candidate object for theauthentication sequence, a physical object belonging to an object typedetermined based on a preference of the user among spatial informationcorresponding to a physical space where the user is positioned.
 18. Theelectronic device of claim 1, wherein the display circuit comprises: anoptical see-through display circuit configured to transmit lightreflected from a physical object in the real world to eyes of the userand provide image information generated based on a virtual object, and avideo see-through display circuit configured to provide imageinformation generated based on the physical object and virtual objectcaptured by a camera.
 19. A method implemented by a processor, themethod comprising: obtaining current spatial information on a physicalspace where a user is positioned using a sensor configured to collectinformation for providing augmented reality; generating a greater numberof virtual objects than the number of unavailable authenticationobjects, based on at least one authentication object among a pluralityof authentication objects being unavailable in the current spatialinformation; outputting the generated virtual objects to a plurality ofpositions comprising a physical position of the unavailableauthentication object using a display circuit configured to provide theaugmented reality; and determining that authentication is successfulbased on a user input that selects a generated virtual object, of thegenerated virtual objects, and an available authentication object as aregistered sequence.
 20. The method of claim 19, further comprisingdetermining that the at least one authentication object is unavailablebased on at least one of: determining that the at least oneauthentication object is not detected, determining that a state of theat least one authentication object has changed, determining that aposition of the at least one authentication object has changed, ordetermining that a shape of the at least one authentication object haschanged.